Tom's Wonderland

Config

Load shared libraries directly from local folder, without any git commit or push

As we know, to interact with webservices, there are multiple ways to do in Ansible realm.

I am not sure is there anything like this for ansible: I need a global shared handlers, which I can call anywhere I want, ie, from playbook, from any roles inside. I can’t find any official document or tutorial anywhere regarding my use case, so I have to cook up myself, and share here. Hopefully it can help someone who need it.

It has been quite a while since last post, as I move to a new company where I can’t easily sync anything to github due to strict security policy, of course, there is nothing relative any policy here. So I have to type something all from my memory. Oh, God.

S3 upload doesn’t work all of a sudden

So, in the morning, we have a incident reported that our java app based on AWS S3 suddenly ceases to work with error:

Mr. Jenkins caught in trouble

Mr. Jenkins reports he can’t finish his task recently because of that :

Well, you may run into scenarios like that sooner or later: it’s well known how to gitlab as a whole, but how to restore gitlab db ONLY ?

As we know, git is not designed to deal with large file storage(LFS), that is why git lfs comes in.

Sometimes, we hope to find out those jobs started by timer , and get info from it

Say we want to develop chef cookbook locally or we only have very few nodes to appy chef cookbook, so we don’t need dedicted chef server. So how ?

HTTP 403 Forbidden

Hereby I wish I could demostrate how to enable verbose logging for gitlab database issue.

So, when trying to execute sudo -E docker, it takes for ever to show.

chef-shell is very helpful tool for testing/debuging code. Let’s show you with a example.

Today, after restart gitlab with gitlab-ctl start, redis does not come up.

Built-in way

It’s very interesting to manipulate gitlab object directly with gitlab console.

Issue with Berkshelf

One simple demo on how to reuse serverspec test cases with shared_examples

Say we have a template resource as

Just a good example of yaml reuse from here

Attributes from .kitchen.yml is not designed to load for chefspec

In my test environment, our product ldap server is inaccessible due to security issues. But, we need to verify ldap login for gitlab. So, we introduce a simple ldap server setup here.

Data Bag Issue

In general, we depend on .kitchen.yml to spin up machine to test chef cookbook, it’s just ok in most case.

I am looking for script like this for a long time, but could not find anyway, so I post here, hopefully it’s useful for someone.

List Chef Config of Current Enviroment

you can choose to display different env/config with or without -c /etc/chef/client.rb ```console #=> knife exec -c /etc/chef/client.rb -E ‘Chef::Config.keys.each { |c| puts “#{c.to_s} : #{Chef::Config[c.to_sym]}” }’

As well known, the format of here-documents is:

I was paged by monitoring team yesterday, claiming paging space issue.

So, that comes a need to create cron job dynamically on inactive server in order to save load. And, our active/inactive pair are swaping from node to node due to failover, so I hope to come up a automatic way to create cron job dymacially.

I am seeing chef backup failure this morning, which gave me error like:

ERROR: Network Error: Error connecting to https://chef.example.com/organizations/example/cookbooks/example-cookbook/1.1.1 - getaddrinfo: Temporary failure in name resolution

It looks odd to me, when I login and try to ping and nslookup chef.example.com, it just works fine.

If LDAP login is not working for you, try following steps to debug.

I got a weird error while converging gitlab, which looks like:

So, again, let’s review how to use postgresql_access resource of postgres 7.1.1 cookbook.

Also, we decide to upgrade postgres installation as well, so I found the latest postgresql cookbook 7.1.1 here.

Ok,it’s time to upgrade your gitlab server, and it’s 11.3.5 on gitlab now.

Well, for our chef client, we are seeing continuously error like below, that is killing out chef run.

error: rpmdb: BDB0113 Thread/process 4149/140556735592512 failed: BDB1507 Thread died Berkeley DB library
error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery
error: cannot open Packages index using db5 -  (-30973)
error: cannot open Packages database in /var/lib/rpm
error: rpmdb: BDB0113 Thread/process 4149/140556735592512 failed: BDB1507 Thread died in Berkeley DB library
error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery
error: cannot open Packages database in /var/lib/rpm

Disconnect slave from master for maintenance.

If you found some jenkins jobs are stuck in hours without exit, you probally have a try with following script:

Validatorless Bootstrap

I believe there can be hundreds or thousands of articles over the internet explaining the bootstrap process of chef, but after reading them, I still have doubt about that, so I decide to look into it in the code level.

Interesting Fact