Model: SoLU Model: 2 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 688 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-2l')

Text #0

Max Range: 0.1151. Min Range: -0.1151

Max Act: 0.1151. Min Act: -0.0003

Data Index: 885457 (C4 (Web Text))

Max Activating Token Index: 816

Text #1

Max Range: 0.1151. Min Range: -0.1151

Max Act: 0.1064. Min Act: -0.0003

Data Index: 338607 (C4 (Web Text))

Max Activating Token Index: 149

Text #2

Max Range: 0.1151. Min Range: -0.1151

Max Act: 0.0958. Min Act: -0.0004

Data Index: 1012155 (C4 (Web Text))

Max Activating Token Index: 408

Text #3

Max Range: 0.1151. Min Range: -0.1151

Max Act: 0.0978. Min Act: -0.0003

Data Index: 712916 (C4 (Web Text))

Max Activating Token Index: 620

Text #4

Max Range: 0.1151. Min Range: -0.1151

Max Act: 0.0960. Min Act: -0.0003

Data Index: 183401 (C4 (Web Text))

Max Activating Token Index: 358

Text #5

Max Range: 0.1151. Min Range: -0.1151

Max Act: 0.0973. Min Act: -0.0003

Data Index: 620646 (C4 (Web Text))

Max Activating Token Index: 684

Text #6

Max Range: 0.1151. Min Range: -0.1151

Max Act: 0.0907. Min Act: -0.0003

Data Index: 587120 (C4 (Web Text))

Max Activating Token Index: 315

Text #7

Max Range: 0.1151. Min Range: -0.1151

Max Act: 0.0910. Min Act: -0.0003

Data Index: 772318 (C4 (Web Text))

Max Activating Token Index: 184

Text #8

Max Range: 0.1151. Min Range: -0.1151

Max Act: 0.0890. Min Act: -0.0003

Data Index: 233874 (C4 (Web Text))

Max Activating Token Index: 258

Text #9

Max Range: 0.1151. Min Range: -0.1151

Max Act: 0.0885. Min Act: -0.0003

Data Index: 1251086 (C4 (Web Text))

Max Activating Token Index: 286

Text #10

Max Range: 0.1151. Min Range: -0.1151

Max Act: 0.0872. Min Act: -0.0003

Data Index: 828550 (C4 (Web Text))

Max Activating Token Index: 82

Text #11

Max Range: 0.1151. Min Range: -0.1151

Max Act: 0.0884. Min Act: -0.0004

Data Index: 830232 (C4 (Web Text))

Max Activating Token Index: 843

Text #12

Max Range: 0.1151. Min Range: -0.1151

Max Act: 0.0822. Min Act: -0.0003

Data Index: 1268864 (C4 (Web Text))

Max Activating Token Index: 702

Text #13

Max Range: 0.1151. Min Range: -0.1151

Max Act: 0.0842. Min Act: -0.0003

Data Index: 486048 (C4 (Web Text))

Max Activating Token Index: 31

Text #14

Max Range: 0.1151. Min Range: -0.1151

Max Act: 0.0846. Min Act: -0.0003

Data Index: 1149480 (C4 (Web Text))

Max Activating Token Index: 509

Text #15

Max Range: 0.1151. Min Range: -0.1151

Max Act: 0.0843. Min Act: -0.0003

Data Index: 684462 (C4 (Web Text))

Max Activating Token Index: 197

Text #16

Max Range: 0.1151. Min Range: -0.1151

Max Act: 0.0823. Min Act: -0.0003

Data Index: 1097458 (C4 (Web Text))

Max Activating Token Index: 526

Text #17

Max Range: 0.1151. Min Range: -0.1151

Max Act: 0.0825. Min Act: -0.0003

Data Index: 797096 (C4 (Web Text))

Max Activating Token Index: 782

Text #18

Max Range: 0.1151. Min Range: -0.1151

Max Act: 0.0836. Min Act: -0.0003

Data Index: 324944 (C4 (Web Text))

Max Activating Token Index: 366

Text #19

Max Range: 0.1151. Min Range: -0.1151

Max Act: 0.0798. Min Act: -0.0003

Data Index: 588803 (C4 (Web Text))

Max Activating Token Index: 637