Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4384 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.9332. Min Range: -0.9332

Max Act: 0.9332. Min Act: -0.0000

Data Index: 174832 (C4 (Web Text))

Max Activating Token Index: 710

Text #1

Max Range: 0.9332. Min Range: -0.9332

Max Act: 0.8092. Min Act: -0.0000

Data Index: 1013513 (C4 (Web Text))

Max Activating Token Index: 470

Text #2

Max Range: 0.9332. Min Range: -0.9332

Max Act: 0.6843. Min Act: -0.0000

Data Index: 997272 (C4 (Web Text))

Max Activating Token Index: 793

Text #3

Max Range: 0.9332. Min Range: -0.9332

Max Act: 0.6917. Min Act: -0.0000

Data Index: 1230151 (C4 (Web Text))

Max Activating Token Index: 408

Text #4

Max Range: 0.9332. Min Range: -0.9332

Max Act: 0.6463. Min Act: -0.0000

Data Index: 21202 (C4 (Web Text))

Max Activating Token Index: 967

Text #5

Max Range: 0.9332. Min Range: -0.9332

Max Act: 0.6598. Min Act: -0.0000

Data Index: 1194138 (C4 (Web Text))

Max Activating Token Index: 1017

Text #6

Max Range: 0.9332. Min Range: -0.9332

Max Act: 0.6894. Min Act: -0.0000

Data Index: 1107984 (C4 (Web Text))

Max Activating Token Index: 637

Text #7

Max Range: 0.9332. Min Range: -0.9332

Max Act: 0.6954. Min Act: -0.0000

Data Index: 968592 (C4 (Web Text))

Max Activating Token Index: 380

Text #8

Max Range: 0.9332. Min Range: -0.9332

Max Act: 0.6756. Min Act: -0.0000

Data Index: 220513 (C4 (Web Text))

Max Activating Token Index: 525

Text #9

Max Range: 0.9332. Min Range: -0.9332

Max Act: 0.6372. Min Act: -0.0000

Data Index: 785370 (C4 (Web Text))

Max Activating Token Index: 775

Text #10

Max Range: 0.9332. Min Range: -0.9332

Max Act: 0.6928. Min Act: -0.0000

Data Index: 320781 (C4 (Web Text))

Max Activating Token Index: 513

Text #11

Max Range: 0.9332. Min Range: -0.9332

Max Act: 0.6571. Min Act: -0.0000

Data Index: 1431033 (Python Code)

Max Activating Token Index: 946

Text #12

Max Range: 0.9332. Min Range: -0.9332

Max Act: 0.6020. Min Act: -0.0000

Data Index: 812319 (C4 (Web Text))

Max Activating Token Index: 587

Text #13

Max Range: 0.9332. Min Range: -0.9332

Max Act: 0.5207. Min Act: -0.0000

Data Index: 1261999 (C4 (Web Text))

Max Activating Token Index: 1010

Text #14

Max Range: 0.9332. Min Range: -0.9332

Max Act: 0.6083. Min Act: -0.0000

Data Index: 1051030 (C4 (Web Text))

Max Activating Token Index: 489

Text #15

Max Range: 0.9332. Min Range: -0.9332

Max Act: 0.6025. Min Act: -0.0000

Data Index: 567454 (C4 (Web Text))

Max Activating Token Index: 687

Text #16

Max Range: 0.9332. Min Range: -0.9332

Max Act: 0.6192. Min Act: -0.0000

Data Index: 285335 (C4 (Web Text))

Max Activating Token Index: 550

Text #17

Max Range: 0.9332. Min Range: -0.9332

Max Act: 0.5991. Min Act: -0.0000

Data Index: 910852 (C4 (Web Text))

Max Activating Token Index: 706

Text #18

Max Range: 0.9332. Min Range: -0.9332

Max Act: 0.5924. Min Act: -0.0000

Data Index: 1165795 (C4 (Web Text))

Max Activating Token Index: 467

Text #19

Max Range: 0.9332. Min Range: -0.9332

Max Act: 0.5925. Min Act: -0.0000

Data Index: 619332 (C4 (Web Text))

Max Activating Token Index: 338