Model: SoLU Model: 6 Layers, 3072 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 75 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-6l')

Text #0

Max Range: 0.1888. Min Range: -0.1888

Max Act: 0.1888. Min Act: -0.0001

Data Index: 554054 (C4 (Web Text))

Max Activating Token Index: 210

Text #1

Max Range: 0.1888. Min Range: -0.1888

Max Act: 0.1697. Min Act: -0.0001

Data Index: 497408 (C4 (Web Text))

Max Activating Token Index: 230

Text #2

Max Range: 0.1888. Min Range: -0.1888

Max Act: 0.1554. Min Act: -0.0001

Data Index: 489081 (C4 (Web Text))

Max Activating Token Index: 478

Text #3

Max Range: 0.1888. Min Range: -0.1888

Max Act: 0.1570. Min Act: -0.0001

Data Index: 429507 (C4 (Web Text))

Max Activating Token Index: 426

Text #4

Max Range: 0.1888. Min Range: -0.1888

Max Act: 0.1561. Min Act: -0.0001

Data Index: 370353 (C4 (Web Text))

Max Activating Token Index: 423

Text #5

Max Range: 0.1888. Min Range: -0.1888

Max Act: 0.1486. Min Act: -0.0001

Data Index: 1336204 (C4 (Web Text))

Max Activating Token Index: 397

Text #6

Max Range: 0.1888. Min Range: -0.1888

Max Act: 0.1450. Min Act: -0.0001

Data Index: 1187940 (C4 (Web Text))

Max Activating Token Index: 458

Text #7

Max Range: 0.1888. Min Range: -0.1888

Max Act: 0.1390. Min Act: -0.0001

Data Index: 116312 (C4 (Web Text))

Max Activating Token Index: 559

Text #8

Max Range: 0.1888. Min Range: -0.1888

Max Act: 0.1403. Min Act: -0.0001

Data Index: 477090 (C4 (Web Text))

Max Activating Token Index: 319

Text #9

Max Range: 0.1888. Min Range: -0.1888

Max Act: 0.1221. Min Act: -0.0001

Data Index: 676603 (C4 (Web Text))

Max Activating Token Index: 899

Text #10

Max Range: 0.1888. Min Range: -0.1888

Max Act: 0.0963. Min Act: -0.0001

Data Index: 699617 (C4 (Web Text))

Max Activating Token Index: 610

Text #11

Max Range: 0.1888. Min Range: -0.1888

Max Act: 0.1326. Min Act: -0.0001

Data Index: 527507 (C4 (Web Text))

Max Activating Token Index: 234

Text #12

Max Range: 0.1888. Min Range: -0.1888

Max Act: 0.1305. Min Act: -0.0001

Data Index: 605861 (C4 (Web Text))

Max Activating Token Index: 257

Text #13

Max Range: 0.1888. Min Range: -0.1888

Max Act: 0.1271. Min Act: -0.0001

Data Index: 1051677 (C4 (Web Text))

Max Activating Token Index: 436

Text #14

Max Range: 0.1888. Min Range: -0.1888

Max Act: 0.1165. Min Act: -0.0001

Data Index: 113380 (C4 (Web Text))

Max Activating Token Index: 564

Text #15

Max Range: 0.1888. Min Range: -0.1888

Max Act: 0.1349. Min Act: -0.0001

Data Index: 738958 (C4 (Web Text))

Max Activating Token Index: 440

Text #16

Max Range: 0.1888. Min Range: -0.1888

Max Act: 0.1265. Min Act: -0.0001

Data Index: 898894 (C4 (Web Text))

Max Activating Token Index: 891

Text #17

Max Range: 0.1888. Min Range: -0.1888

Max Act: 0.1131. Min Act: -0.0001

Data Index: 275158 (C4 (Web Text))

Max Activating Token Index: 681

Text #18

Max Range: 0.1888. Min Range: -0.1888

Max Act: 0.1222. Min Act: -0.0001

Data Index: 462828 (C4 (Web Text))

Max Activating Token Index: 171

Text #19

Max Range: 0.1888. Min Range: -0.1888

Max Act: 0.1211. Min Act: -0.0001

Data Index: 1100281 (C4 (Web Text))

Max Activating Token Index: 874