Reading a JSON file, adding to it and then saving it

Archives Ansible Project
1

Hi,

I’m trying to add lines to the Logstash Forwarder config file, which looks like this right now:

{
“network”: {
“servers”: [ “1.2.3.4:5000” ],
“timeout”: 15,
“ssl ca”: “/etc/pki/tls/certs/logstash-forwarder.crt”
},
“files”: [
{
“paths”: [
“/logs/access.log”
],
“fields”: { “type”: “web_nginx_access” }
}
]
}

And I’d like to add other logs in that files.paths array, then save it. The problem is that starting from the second line I will have to add the commas to not screw up the JSON syntax… how can I do that with Ansible.
I’ve tried so many solutions, just can’t figure this one out.

I was thinking about using a “command cat logstash-forwarder.conf” plus “register: result” then somehow manipulating the object with {{ result.stdout|from_json }} but that’s as far as I got. I’d really appreciate any comments.

Thanks,
Mark

2

Rather than doing this in a single logstash config file, I would suggest using a templated version of this file. For every log file that you want logstash to monitor, you can drop a separate file into /etc/logstash/conf.d/ and it will pick them all up (if you are using the default setting for logstash).

3

I don’t believe logstash forwarder supports conf.d type configuration yet - https://github.com/elasticsearch/logstash-forwarder/issues/244

I’m curious how others have assembled a config file when a seerver is a member of multiple groups

thanks,

Joe

4

My bad. I was assuming that logstash-forwarder had the same feature that regular logstash did.

It’s kludgy, but how about having each group drop a list into a file on the target system, and then have a handler that would assemble the files into the final json configuration?

Also, according to the logstash-forwarder documentation, you can use a file glob, so could you put your log files into a single directory and use *.log or something like that?

5

Mark G.,

The to_json (or to_nice_json) filter and a template will do the magic you want. Given this playbook:

`

  • hosts: localhost
    connection: local
    vars:
    log_paths:
  • “/logs/access.log”
  • “/logs/error.log”
    tasks:
  • name: create config file
    template: src=logstash.json.j2 dest=/tmp/logstash.json
    `

and this template:

`
{
“network”: {
“servers”: [ “1.2.3.4:5000” ],
“timeout”: 15,
“ssl ca”: “/etc/pki/tls/certs/logstash-forwarder.crt”
},
“files”: [
{
“paths”: {{ log_paths | to_json }},
“fields”: { “type”: “web_nginx_access” }
}
]
}

`

Will render:

`
{
“network”: {
“servers”: [ “1.2.3.4:5000” ],
“timeout”: 15,
“ssl ca”: “/etc/pki/tls/certs/logstash-forwarder.crt”
},
“files”: [
{
“paths”: [“/logs/access.log”, “/logs/error.log”],
“fields”: { “type”: “web_nginx_access” }
}
]
}

`

  • James
6

Thanks James and everyone, the other night I ended up doing almost exactly like what you suggested and it works great!

Cheers,
Mark