INDEX 0MY TIME TO SHINE!
It was three weeks ago. I had spent the entire day being ruthlessly defeated by LSTMs in the battery Remaining Useful Life (RUL) prediction project. Just as I was ready to embrace the day's defeat, a notification caught my eye: Localised Precipitation Forecasting in Brazzaville Using AI, a Zindi competition. This opportunity was exactly what I needed to get my hands a bit dirty.
As a beginner armed with little more than basic ML knowledge I was skeptical. But as a lifelong enjoyer of learning (and frequent victim of curiosity-fueled rabbit holes), something about it felt… magnetic. That evening, with a reckless blend of naivety and determination and a stubborn refusal to quit. I decided it was my time to shine!
Spoiler: I did (in fact) not shine.
[INFO] Loading enthusiasm... 100%
[INFO] Loading domain knowledge... 15%
[WARNING] Overfitting detected in ego layer
INDEX 1RAIN-DANCE
Tools: pycaret, pandas, numpy, seaborn, matplotlib.pyplot, sklearn.metrics.mean_squared_error, zipfile
Before diving into my (mis)adventures, let's rewind to the rules of the game. The Zindi challenge demanded an open-source ML model to predict precipitation in Brazzaville. Erratic rainfall, characterized by sudden shifts between droughts and floods, adversely affects infrastructure and livelihoods. This unpredictable weather pattern creates significant challenges for communities, making it difficult to sustain essential services and economic stability.
- Goal: Predict daily/sub-daily rainfall using only the provided datasets (no external data allowed).
- Rules: Open-source tools only (RIP, GPT-4), teams of ≤4, and a 100-submission limit, 5 per day.
- Evaluation: RMSE on a 70% private test set.
A perfect storm for a beginner's first rodeo.
Try Your Own RMSE Calculation
Enter predicted and actual values to see how RMSE works:
With the rules carved into my soul (and a fresh Jupyter notebook ominously named raindance.ipynb), I began my ritual:
Data Archaeology:
The provided datasets were a mix of historical rainfall measurements, atmospheric variables (temperature, humidity, etc.), and temporal markers (dates, seasons). The main issue with the data was the outliers.
My move: A combination of pandas_profiling, histograms, and correlation checks and a lot of caffeine.
from pandas_profiling import ProfileReport
# Load the data
df = pd.read_csv('rainfall_data.csv')
# Generate EDA report
profile = ProfileReport(df, title='Rainfall EDA')
profile.to_file("rainfall_eda.html")
# Basic outlier detection
df = df[(df['precipitation'] >= 0) & (df['precipitation'] <= 200)]
Rapid Prototyping:
Baseline Models: The basic models from pycaret where used first using the default hyperparameters.
Upgrade Attempt: hyperparameter tuning was introduced
Hail Mary: some feature engineering and a validation split
Data ≠ Truth: The provided atmospheric variables had gaps wider than my understanding of time-series cross-validation.
Feature Engineering is King: The top performers (as I'd later learn) didn't just use the data they transformed it into meteorological hieroglyphics I couldn't yet decipher.
By the end of raindance.ipynb, I had two things:
- A model that was good at predicting normal data but the outlier situation was making it very bad.
- And a comfortable leaderboard position of 4 (but to be fair only a handful of people had joined let alone worked on the competition)
- Ideas! More feature engineering, trying ensembles, a multistage prediction approach where I would first predict if rain occurs at all before I estimate how much of it occurs.
| Rank | Team | RMSE | Models |
|---|---|---|---|
| 1 | WeatherWizards | 12.34 | XGBoost |
| 2 | RainChasers | 13.45 | LightGBM |
| 3 | Precipitators | 14.56 | CatBoost |
| 4 | Me | 15.67 | Random Forest |
| 5 | CloudSeekers | 16.78 | Linear Reg |
INDEX 2Quentin Tarantino Level of Temporal Confusion
The next day, I found myself pushed out of the top 10 as more competitive solutions flooded the leaderboard. Determined to regain ground, I turned my attention to the temporal aspects of the data, convinced that leveraging time-series modeling would give me an edge. I assumed that an LSTM or RNN, properly configured, would capture the sequential nature of rainfall patterns and improve predictions.
LSTM Performance Confusion Matrix
| TP: 42 | FP: 28 |
| FN: 35 | TN: 45 |
Accuracy: 58% (Not great)
I implemented an LSTM, trying to take advantage of some temporal order, training on earlier data and validating on later data. However, the results were disastrous (on the test data). The model performed worse. Confused, I took a step back and scrutinized the data split methodology. That was when I realized the critical drawback: the competition's train-test split was random, not chronological.
from keras.models import Sequential
from keras.layers import LSTM, Dense
# Create LSTM model
model = Sequential()
model.add(LSTM(50, activation='relu', input_shape=(n_steps, n_features)))
model.add(Dense(1))
model.compile(optimizer='adam', loss='mse')
# Train model (on randomly split data - oops!)
model.fit(X_train, y_train, epochs=100, verbose=0)
This meant that any attempt to impose a temporal structure on training was fundamentally flawed. The LSTM, designed to learn from sequences, was instead being fed shuffled data points with no meaningful temporal relationship. The result was a model that couldn't generalize, trapped in what I now thought of as a Quentin Tarantino-level of temporal confusion—where past, present, and future were jumbled beyond recognition.
By the end of this experiment, I had no improvement in my leaderboard score, only a hard-earned lesson: without a time-ordered split, traditional time-series models were useless. I abandoned the LSTM approach and returned to more robust methods, now with a clearer understanding of the competition's constraints.
INDEX 3EUREKA!
On the third day, I shifted focus entirely to feature engineering and hyperparameter tuning. I discarded the failed time-series approach and instead concentrated on refining my model. Through systematic trial and error I finally stumbled upon a combination that worked.
| Parameter | Initial Value | Tuned Value | Impact |
|---|---|---|---|
| n_estimators | 100 | 500 | +2% |
| max_depth | None | 7 | +1.5% |
| learning_rate | 0.1 | 0.05 | +1% |
| subsample | 1.0 | 0.8 | +0.5% |
My score jumped to first place on the leaderboard. For a brief moment, I enjoyed having my name at the top. But I knew it wouldn't last. I had reached the limits of my current understanding.
| Rank | Team | RMSE | Models |
|---|---|---|---|
| 1 | Me | 12.34 | XGBoost Tuned |
| 2 | WeatherWizards | 12.45 | XGBoost |
| 3 | RainChasers | 13.56 | LightGBM |
Recognizing that I needed deeper knowledge to sustain my lead, I decided to pause submissions and study advanced regression techniques. I bookmarked papers, revisited feature selection methods, and planned my next move. But as life often does, other priorities took over. Days turned into weeks, and before I knew it, the competition had closed.
When the final rankings were revealed, I found myself in 22nd place. A respectable position, but a stark reminder of how quickly progress moves in machine learning. My brief stint at the top had been overtaken by those who either had more domain expertise, better models, or more simply THE BETTER MAN. Still, I walked away with invaluable lessons: feature engineering matters, consistency is key, and in AI competitions, standing still means falling behind.
INDEX 4THE BETTER MAN PART 1
After the competition ended, I dug into the first-place solution. A clean, MLOps-ready pipeline that felt worlds apart from my own hastily-evolving notebook. What stood out first was the structure: separate scripts for data loading, preprocessing, model training, utilities, and a command-line interface for training and inference. It wasn't just a model; it was a system.
precipitation-forecast/
├── data/ # Raw and processed data
├── features/ # Feature engineering
│ ├── temporal.py
│ └── atmospheric.py
├── models/ # Model definitions
│ ├── catboost.py
│ └── lightgbm.py
├── train.py # Training script
├── predict.py # Inference script
└── utils/ # Helper functions
├── metrics.py
└── visualization.py
Feature engineering played a massive role with lag and lead precipitation values, first-order differences in humidity and pressure, and other temporal features that gave the models "memory" of recent weather patterns. The winner didn't rely on a single model either. They trained both CatBoost and LightGBM, each with cross-validation strategies designed to avoid temporal leakage, then stacked their predictions with a Ridge regression meta-learner.
The key difference between their approach and mine was strategic scope. While I focused narrowly on tuning a single model, they optimized the entire workflow in a way that made iteration fast and robust. They also respected the time-series nature of the data from the start, something I learned the hard way.
Studying their pipeline made me realize that winning solutions are about thoughtful feature engineering, multiple complementary models, and infrastructure that makes experimentation easy. I might have finished 22nd, but I walked away with a blueprint for how to approach my next competition.
INDEX 4THE BETTER MAN PART 3
The third-place solution took a methodical approach to feature engineering that revealed clear gaps in my own workflow. While I had briefly experimented with lag features, my implementation didn't yield meaningful gains, and I abandoned them early. This competitor, however, implemented a systematic temporal encoding strategy that captured both short-term weather memory and seasonal cycles. Their lag features at intervals of [1, 2, 3, 4, 7, 21, 30] days for precipitation and eight key atmospheric variables created a detailed climate fingerprint of Brazzaville. The inclusion of 21- and 30-day lags particularly impressed me — these longer windows captured monthly cycles I hadn't properly leveraged.
Feature Engineering Simulator
Select features to include in your model:
Another strength was their disciplined feature selection pipeline. Where I kept most engineered features (risking noise), they aggressively pruned using three steps: removing constant features, dropping highly correlated variables (threshold 0.8), and eliminating duplicates. Their cyclical encoding of datetime features (month, quarter, week, etc.) preserved temporal relationships that my own encoding sometimes lost when I relied on one-hot representations.
def remove_correlated_features(df, threshold=0.8):
corr_matrix = df.corr().abs()
upper = corr_matrix.where(np.triu(np.ones(corr_matrix.shape), k=1).astype(bool)
to_drop = [column for column in upper.columns if any(upper[column] > threshold)]
return df.drop(to_drop, axis=1)
# Cyclical encoding for datetime features
df['month_sin'] = np.sin(2 * np.pi * df['month']/12)
df['month_cos'] = np.cos(2 * np.pi * df['month']/12)
The key lesson here was balance. Creativity in feature generation paired with rigorous filtering. While I focused heavily on model experimentation, they invested in building a clean, domain-aware feature set that made their models inherently stronger. In weather forecasting competitions, this preprocessing discipline can matter more than model choice itself, a perspective I only truly appreciated after the competition.
1. Feature engineering often beats model tuning
2. A well-structured pipeline enables faster iteration
3. Understanding the evaluation setup is crucial
4. Even "failed" competitions provide valuable learning
5. The journey from 22nd to 1st is just more feature engineering