Model: SoLU Model: 4 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 846 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-4l')

Text #0

Max Range: 0.1183. Min Range: -0.1183

Max Act: 0.1183. Min Act: -0.0005

Data Index: 1236165 (C4 (Web Text))

Max Activating Token Index: 391

Text #1

Max Range: 0.1183. Min Range: -0.1183

Max Act: 0.1144. Min Act: -0.0005

Data Index: 408608 (C4 (Web Text))

Max Activating Token Index: 21

Text #2

Max Range: 0.1183. Min Range: -0.1183

Max Act: 0.1121. Min Act: -0.0005

Data Index: 318812 (C4 (Web Text))

Max Activating Token Index: 485

Text #3

Max Range: 0.1183. Min Range: -0.1183

Max Act: 0.1075. Min Act: -0.0005

Data Index: 404369 (C4 (Web Text))

Max Activating Token Index: 514

Text #4

Max Range: 0.1183. Min Range: -0.1183

Max Act: 0.1076. Min Act: -0.0005

Data Index: 942081 (C4 (Web Text))

Max Activating Token Index: 470

Text #5

Max Range: 0.1183. Min Range: -0.1183

Max Act: 0.1035. Min Act: -0.0005

Data Index: 955283 (C4 (Web Text))

Max Activating Token Index: 528

Text #6

Max Range: 0.1183. Min Range: -0.1183

Max Act: 0.1064. Min Act: -0.0005

Data Index: 528790 (C4 (Web Text))

Max Activating Token Index: 771

Text #7

Max Range: 0.1183. Min Range: -0.1183

Max Act: 0.1055. Min Act: -0.0005

Data Index: 7020 (C4 (Web Text))

Max Activating Token Index: 82

Text #8

Max Range: 0.1183. Min Range: -0.1183

Max Act: 0.1037. Min Act: -0.0005

Data Index: 681836 (C4 (Web Text))

Max Activating Token Index: 79

Text #9

Max Range: 0.1183. Min Range: -0.1183

Max Act: 0.1017. Min Act: -0.0005

Data Index: 820511 (C4 (Web Text))

Max Activating Token Index: 179

Text #10

Max Range: 0.1183. Min Range: -0.1183

Max Act: 0.1010. Min Act: -0.0005

Data Index: 679897 (C4 (Web Text))

Max Activating Token Index: 872

Text #11

Max Range: 0.1183. Min Range: -0.1183

Max Act: 0.0990. Min Act: -0.0005

Data Index: 270099 (C4 (Web Text))

Max Activating Token Index: 632

Text #12

Max Range: 0.1183. Min Range: -0.1183

Max Act: 0.0981. Min Act: -0.0005

Data Index: 59566 (C4 (Web Text))

Max Activating Token Index: 111

Text #13

Max Range: 0.1183. Min Range: -0.1183

Max Act: 0.1010. Min Act: -0.0005

Data Index: 1019881 (C4 (Web Text))

Max Activating Token Index: 977

Text #14

Max Range: 0.1183. Min Range: -0.1183

Max Act: 0.0902. Min Act: -0.0005

Data Index: 1023928 (C4 (Web Text))

Max Activating Token Index: 454

Text #15

Max Range: 0.1183. Min Range: -0.1183

Max Act: 0.0925. Min Act: -0.0005

Data Index: 577866 (C4 (Web Text))

Max Activating Token Index: 561

Text #16

Max Range: 0.1183. Min Range: -0.1183

Max Act: 0.1043. Min Act: -0.0005

Data Index: 592611 (C4 (Web Text))

Max Activating Token Index: 556

Text #17

Max Range: 0.1183. Min Range: -0.1183

Max Act: 0.0941. Min Act: -0.0005

Data Index: 1431516 (Python Code)

Max Activating Token Index: 588

Text #18

Max Range: 0.1183. Min Range: -0.1183

Max Act: 0.1034. Min Act: -0.0005

Data Index: 1129950 (C4 (Web Text))

Max Activating Token Index: 532

Text #19

Max Range: 0.1183. Min Range: -0.1183

Max Act: 0.0985. Min Act: -0.0005

Data Index: 903767 (C4 (Web Text))

Max Activating Token Index: 1019