linux from scratch (LFS) is a good guide for
dealing with the various configurations required for many packages.
as you have realised what is required varies a lot and a guide like LFS can
save a lot of time digging.
Doing everything from scratch sounds a bit hairy but to build a light system
based around busybox you really dont need that much (witness TSlinux).
I found crosstools-ng worth investing time in. It will probably be useful if
you want to build a toolchain with exactly the versions matching the TS root fs
since it makes this easy to configure.
It also provides a tool to copy just the required libs from your build env to
the target fs.
After a lot of wasted time I came to the conclusion (like many others) that the
best way is to just bite the bullet and start from scratch. It takes some time
to get to grips with but its time invested not wasted.
Welcome to embedded !