Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5441 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.6864. Min Range: -0.6864

Max Act: 0.6864. Min Act: -0.0000

Data Index: 578876 (C4 (Web Text))

Max Activating Token Index: 140

Text #1

Max Range: 0.6864. Min Range: -0.6864

Max Act: 0.6030. Min Act: -0.0000

Data Index: 716252 (C4 (Web Text))

Max Activating Token Index: 405

Text #2

Max Range: 0.6864. Min Range: -0.6864

Max Act: 0.5922. Min Act: -0.0000

Data Index: 998471 (C4 (Web Text))

Max Activating Token Index: 959

Text #3

Max Range: 0.6864. Min Range: -0.6864

Max Act: 0.5870. Min Act: -0.0000

Data Index: 1041332 (C4 (Web Text))

Max Activating Token Index: 41

Text #4

Max Range: 0.6864. Min Range: -0.6864

Max Act: 0.5917. Min Act: -0.0000

Data Index: 88215 (C4 (Web Text))

Max Activating Token Index: 143

Text #5

Max Range: 0.6864. Min Range: -0.6864

Max Act: 0.5876. Min Act: -0.0000

Data Index: 902351 (C4 (Web Text))

Max Activating Token Index: 751

Text #6

Max Range: 0.6864. Min Range: -0.6864

Max Act: 0.5561. Min Act: -0.0000

Data Index: 1078163 (C4 (Web Text))

Max Activating Token Index: 780

Text #7

Max Range: 0.6864. Min Range: -0.6864

Max Act: 0.5467. Min Act: -0.0000

Data Index: 363301 (C4 (Web Text))

Max Activating Token Index: 71

Text #8

Max Range: 0.6864. Min Range: -0.6864

Max Act: 0.5593. Min Act: -0.0000

Data Index: 613236 (C4 (Web Text))

Max Activating Token Index: 513

Text #9

Max Range: 0.6864. Min Range: -0.6864

Max Act: 0.5648. Min Act: -0.0000

Data Index: 681433 (C4 (Web Text))

Max Activating Token Index: 785

Text #10

Max Range: 0.6864. Min Range: -0.6864

Max Act: 0.5352. Min Act: -0.0000

Data Index: 569348 (C4 (Web Text))

Max Activating Token Index: 590

Text #11

Max Range: 0.6864. Min Range: -0.6864

Max Act: 0.5110. Min Act: -0.0000

Data Index: 1191177 (C4 (Web Text))

Max Activating Token Index: 171

Text #12

Max Range: 0.6864. Min Range: -0.6864

Max Act: 0.5082. Min Act: -0.0000

Data Index: 851025 (C4 (Web Text))

Max Activating Token Index: 351

Text #13

Max Range: 0.6864. Min Range: -0.6864

Max Act: 0.4914. Min Act: -0.0000

Data Index: 471382 (C4 (Web Text))

Max Activating Token Index: 182

Text #14

Max Range: 0.6864. Min Range: -0.6864

Max Act: 0.5200. Min Act: -0.0000

Data Index: 64423 (C4 (Web Text))

Max Activating Token Index: 27

Text #15

Max Range: 0.6864. Min Range: -0.6864

Max Act: 0.4946. Min Act: -0.0000

Data Index: 1291006 (C4 (Web Text))

Max Activating Token Index: 453

Text #16

Max Range: 0.6864. Min Range: -0.6864

Max Act: 0.5122. Min Act: -0.0000

Data Index: 745287 (C4 (Web Text))

Max Activating Token Index: 279

Text #17

Max Range: 0.6864. Min Range: -0.6864

Max Act: 0.4921. Min Act: -0.0000

Data Index: 249842 (C4 (Web Text))

Max Activating Token Index: 316

Text #18

Max Range: 0.6864. Min Range: -0.6864

Max Act: 0.4890. Min Act: -0.0000

Data Index: 350380 (C4 (Web Text))

Max Activating Token Index: 832

Text #19

Max Range: 0.6864. Min Range: -0.6864

Max Act: 0.4852. Min Act: -0.0000

Data Index: 994050 (C4 (Web Text))

Max Activating Token Index: 94