Model: SoLU Model: 6 Layers, 3072 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2914 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-6l')

Text #0

Max Range: 0.1209. Min Range: -0.1209

Max Act: 0.1209. Min Act: -0.0003

Data Index: 1081988 (C4 (Web Text))

Max Activating Token Index: 782

Text #1

Max Range: 0.1209. Min Range: -0.1209

Max Act: 0.1117. Min Act: -0.0003

Data Index: 633141 (C4 (Web Text))

Max Activating Token Index: 454

Text #2

Max Range: 0.1209. Min Range: -0.1209

Max Act: 0.1075. Min Act: -0.0003

Data Index: 1104283 (C4 (Web Text))

Max Activating Token Index: 428

Text #3

Max Range: 0.1209. Min Range: -0.1209

Max Act: 0.1037. Min Act: -0.0003

Data Index: 712408 (C4 (Web Text))

Max Activating Token Index: 47

Text #4

Max Range: 0.1209. Min Range: -0.1209

Max Act: 0.1040. Min Act: -0.0003

Data Index: 323748 (C4 (Web Text))

Max Activating Token Index: 841

Text #5

Max Range: 0.1209. Min Range: -0.1209

Max Act: 0.1064. Min Act: -0.0003

Data Index: 696016 (C4 (Web Text))

Max Activating Token Index: 746

Text #6

Max Range: 0.1209. Min Range: -0.1209

Max Act: 0.1066. Min Act: -0.0003

Data Index: 186294 (C4 (Web Text))

Max Activating Token Index: 197

Text #7

Max Range: 0.1209. Min Range: -0.1209

Max Act: 0.1069. Min Act: -0.0003

Data Index: 881989 (C4 (Web Text))

Max Activating Token Index: 203

Text #8

Max Range: 0.1209. Min Range: -0.1209

Max Act: 0.1111. Min Act: -0.0003

Data Index: 1107020 (C4 (Web Text))

Max Activating Token Index: 513

Text #9

Max Range: 0.1209. Min Range: -0.1209

Max Act: 0.0925. Min Act: -0.0003

Data Index: 480501 (C4 (Web Text))

Max Activating Token Index: 494

Text #10

Max Range: 0.1209. Min Range: -0.1209

Max Act: 0.0990. Min Act: -0.0003

Data Index: 65410 (C4 (Web Text))

Max Activating Token Index: 342

Text #11

Max Range: 0.1209. Min Range: -0.1209

Max Act: 0.0975. Min Act: -0.0003

Data Index: 508096 (C4 (Web Text))

Max Activating Token Index: 21

Text #12

Max Range: 0.1209. Min Range: -0.1209

Max Act: 0.0966. Min Act: -0.0003

Data Index: 757719 (C4 (Web Text))

Max Activating Token Index: 730

Text #13

Max Range: 0.1209. Min Range: -0.1209

Max Act: 0.0967. Min Act: -0.0003

Data Index: 1106713 (C4 (Web Text))

Max Activating Token Index: 496

Text #14

Max Range: 0.1209. Min Range: -0.1209

Max Act: 0.0964. Min Act: -0.0003

Data Index: 1203943 (C4 (Web Text))

Max Activating Token Index: 189

Text #15

Max Range: 0.1209. Min Range: -0.1209

Max Act: 0.0953. Min Act: -0.0003

Data Index: 1288977 (C4 (Web Text))

Max Activating Token Index: 252

Text #16

Max Range: 0.1209. Min Range: -0.1209

Max Act: 0.0920. Min Act: -0.0003

Data Index: 1247701 (C4 (Web Text))

Max Activating Token Index: 975

Text #17

Max Range: 0.1209. Min Range: -0.1209

Max Act: 0.0992. Min Act: -0.0003

Data Index: 239817 (C4 (Web Text))

Max Activating Token Index: 57

Text #18

Max Range: 0.1209. Min Range: -0.1209

Max Act: 0.0884. Min Act: -0.0003

Data Index: 494294 (C4 (Web Text))

Max Activating Token Index: 703

Text #19

Max Range: 0.1209. Min Range: -0.1209

Max Act: 0.0933. Min Act: -0.0003

Data Index: 172184 (C4 (Web Text))

Max Activating Token Index: 164