Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3132 in Layer 10

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.2498. Min Range: -0.2498

Max Act: 0.2498. Min Act: -0.0000

Data Index: 441858 (C4 (Web Text))

Max Activating Token Index: 887

Text #1

Max Range: 0.2498. Min Range: -0.2498

Max Act: 0.2190. Min Act: -0.0000

Data Index: 686874 (C4 (Web Text))

Max Activating Token Index: 510

Text #2

Max Range: 0.2498. Min Range: -0.2498

Max Act: 0.2202. Min Act: -0.0000

Data Index: 281401 (C4 (Web Text))

Max Activating Token Index: 721

Text #3

Max Range: 0.2498. Min Range: -0.2498

Max Act: 0.1805. Min Act: -0.0000

Data Index: 1302277 (C4 (Web Text))

Max Activating Token Index: 777

Text #4

Max Range: 0.2498. Min Range: -0.2498

Max Act: 0.1731. Min Act: -0.0000

Data Index: 296553 (C4 (Web Text))

Max Activating Token Index: 87

Text #5

Max Range: 0.2498. Min Range: -0.2498

Max Act: 0.1438. Min Act: -0.0000

Data Index: 688745 (C4 (Web Text))

Max Activating Token Index: 326

Text #6

Max Range: 0.2498. Min Range: -0.2498

Max Act: 0.1480. Min Act: -0.0000

Data Index: 1014207 (C4 (Web Text))

Max Activating Token Index: 106

Text #7

Max Range: 0.2498. Min Range: -0.2498

Max Act: 0.1460. Min Act: 0.0000

Data Index: 288929 (C4 (Web Text))

Max Activating Token Index: 222

Text #8

Max Range: 0.2498. Min Range: -0.2498

Max Act: 0.1395. Min Act: -0.0000

Data Index: 277067 (C4 (Web Text))

Max Activating Token Index: 593

Text #9

Max Range: 0.2498. Min Range: -0.2498

Max Act: 0.1448. Min Act: -0.0000

Data Index: 1188757 (C4 (Web Text))

Max Activating Token Index: 625

Text #10

Max Range: 0.2498. Min Range: -0.2498

Max Act: 0.1485. Min Act: -0.0000

Data Index: 510926 (C4 (Web Text))

Max Activating Token Index: 330

Text #11

Max Range: 0.2498. Min Range: -0.2498

Max Act: 0.1318. Min Act: -0.0000

Data Index: 883872 (C4 (Web Text))

Max Activating Token Index: 188

Text #12

Max Range: 0.2498. Min Range: -0.2498

Max Act: 0.1404. Min Act: -0.0000

Data Index: 1104834 (C4 (Web Text))

Max Activating Token Index: 296

Text #13

Max Range: 0.2498. Min Range: -0.2498

Max Act: 0.1341. Min Act: -0.0000

Data Index: 1234325 (C4 (Web Text))

Max Activating Token Index: 1011

Text #14

Max Range: 0.2498. Min Range: -0.2498

Max Act: 0.1357. Min Act: -0.0000

Data Index: 1257144 (C4 (Web Text))

Max Activating Token Index: 507

Text #15

Max Range: 0.2498. Min Range: -0.2498

Max Act: 0.1431. Min Act: -0.0000

Data Index: 1024344 (C4 (Web Text))

Max Activating Token Index: 799

Text #16

Max Range: 0.2498. Min Range: -0.2498

Max Act: 0.1383. Min Act: -0.0000

Data Index: 878759 (C4 (Web Text))

Max Activating Token Index: 932

Text #17

Max Range: 0.2498. Min Range: -0.2498

Max Act: 0.1360. Min Act: -0.0000

Data Index: 928664 (C4 (Web Text))

Max Activating Token Index: 255

Text #18

Max Range: 0.2498. Min Range: -0.2498

Max Act: 0.1271. Min Act: -0.0000

Data Index: 1094063 (C4 (Web Text))

Max Activating Token Index: 204

Text #19

Max Range: 0.2498. Min Range: -0.2498

Max Act: 0.1262. Min Act: -0.0000

Data Index: 897399 (C4 (Web Text))

Max Activating Token Index: 755