Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5747 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 5.1326. Min Range: -5.1326

Max Act: 5.1326. Min Act: -0.0001

Data Index: 495883 (C4 (Web Text))

Max Activating Token Index: 310

Text #1

Max Range: 5.1326. Min Range: -5.1326

Max Act: 5.0137. Min Act: -0.0000

Data Index: 701219 (C4 (Web Text))

Max Activating Token Index: 494

Text #2

Max Range: 5.1326. Min Range: -5.1326

Max Act: 4.9896. Min Act: -0.0001

Data Index: 359314 (C4 (Web Text))

Max Activating Token Index: 337

Text #3

Max Range: 5.1326. Min Range: -5.1326

Max Act: 4.9741. Min Act: -0.0000

Data Index: 318796 (C4 (Web Text))

Max Activating Token Index: 862

Text #4

Max Range: 5.1326. Min Range: -5.1326

Max Act: 4.9457. Min Act: -0.0001

Data Index: 681697 (C4 (Web Text))

Max Activating Token Index: 463

Text #5

Max Range: 5.1326. Min Range: -5.1326

Max Act: 4.6226. Min Act: -0.0000

Data Index: 1020440 (C4 (Web Text))

Max Activating Token Index: 31

Text #6

Max Range: 5.1326. Min Range: -5.1326

Max Act: 4.6265. Min Act: -0.0000

Data Index: 384290 (C4 (Web Text))

Max Activating Token Index: 524

Text #7

Max Range: 5.1326. Min Range: -5.1326

Max Act: 4.6131. Min Act: -0.0000

Data Index: 454293 (C4 (Web Text))

Max Activating Token Index: 373

Text #8

Max Range: 5.1326. Min Range: -5.1326

Max Act: 4.1447. Min Act: -0.0000

Data Index: 810308 (C4 (Web Text))

Max Activating Token Index: 389

Text #9

Max Range: 5.1326. Min Range: -5.1326

Max Act: 4.4754. Min Act: -0.0000

Data Index: 366897 (C4 (Web Text))

Max Activating Token Index: 444

Text #10

Max Range: 5.1326. Min Range: -5.1326

Max Act: 4.2527. Min Act: -0.0001

Data Index: 425233 (C4 (Web Text))

Max Activating Token Index: 782

Text #11

Max Range: 5.1326. Min Range: -5.1326

Max Act: 4.2638. Min Act: -0.0000

Data Index: 9652 (C4 (Web Text))

Max Activating Token Index: 935

Text #12

Max Range: 5.1326. Min Range: -5.1326

Max Act: 4.5872. Min Act: -0.0000

Data Index: 371157 (C4 (Web Text))

Max Activating Token Index: 947

Text #13

Max Range: 5.1326. Min Range: -5.1326

Max Act: 4.6192. Min Act: -0.0000

Data Index: 779768 (C4 (Web Text))

Max Activating Token Index: 853

Text #14

Max Range: 5.1326. Min Range: -5.1326

Max Act: 4.3089. Min Act: -0.0000

Data Index: 298449 (C4 (Web Text))

Max Activating Token Index: 445

Text #15

Max Range: 5.1326. Min Range: -5.1326

Max Act: 4.2867. Min Act: -0.0000

Data Index: 289366 (C4 (Web Text))

Max Activating Token Index: 180

Text #16

Max Range: 5.1326. Min Range: -5.1326

Max Act: 4.4512. Min Act: -0.0000

Data Index: 647967 (C4 (Web Text))

Max Activating Token Index: 877

Text #17

Max Range: 5.1326. Min Range: -5.1326

Max Act: 4.2418. Min Act: -0.0000

Data Index: 1092446 (C4 (Web Text))

Max Activating Token Index: 220

Text #18

Max Range: 5.1326. Min Range: -5.1326

Max Act: 4.2585. Min Act: -0.0000

Data Index: 1173047 (C4 (Web Text))

Max Activating Token Index: 167

Text #19

Max Range: 5.1326. Min Range: -5.1326

Max Act: 3.7969. Min Act: -0.0000

Data Index: 794360 (C4 (Web Text))

Max Activating Token Index: 991