Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 5050 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 0.7764. Min Range: -0.7764

Max Act: 0.7764. Min Act: -0.0000

Data Index: 861723 (The Pile)

Max Activating Token Index: 262

Text #1

Max Range: 0.7764. Min Range: -0.7764

Max Act: 0.6955. Min Act: -0.0000

Data Index: 1896552 (The Pile)

Max Activating Token Index: 694

Text #2

Max Range: 0.7764. Min Range: -0.7764

Max Act: 0.6520. Min Act: -0.0000

Data Index: 605359 (The Pile)

Max Activating Token Index: 546

Text #3

Max Range: 0.7764. Min Range: -0.7764

Max Act: 0.5814. Min Act: -0.0000

Data Index: 194046 (The Pile)

Max Activating Token Index: 128

Text #4

Max Range: 0.7764. Min Range: -0.7764

Max Act: 0.5943. Min Act: -0.0000

Data Index: 1066652 (The Pile)

Max Activating Token Index: 900

Text #5

Max Range: 0.7764. Min Range: -0.7764

Max Act: 0.5229. Min Act: -0.0000

Data Index: 1536310 (The Pile)

Max Activating Token Index: 199

Text #6

Max Range: 0.7764. Min Range: -0.7764

Max Act: 0.5678. Min Act: -0.0000

Data Index: 863123 (The Pile)

Max Activating Token Index: 74

Text #7

Max Range: 0.7764. Min Range: -0.7764

Max Act: 0.5242. Min Act: -0.0000

Data Index: 236202 (The Pile)

Max Activating Token Index: 253

Text #8

Max Range: 0.7764. Min Range: -0.7764

Max Act: 0.5430. Min Act: -0.0000

Data Index: 972164 (The Pile)

Max Activating Token Index: 81

Text #9

Max Range: 0.7764. Min Range: -0.7764

Max Act: 0.5421. Min Act: -0.0000

Data Index: 1558176 (The Pile)

Max Activating Token Index: 98

Text #10

Max Range: 0.7764. Min Range: -0.7764

Max Act: 0.5097. Min Act: -0.0000

Data Index: 426234 (The Pile)

Max Activating Token Index: 94

Text #11

Max Range: 0.7764. Min Range: -0.7764

Max Act: 0.5042. Min Act: -0.0000

Data Index: 232216 (The Pile)

Max Activating Token Index: 97

Text #12

Max Range: 0.7764. Min Range: -0.7764

Max Act: 0.5342. Min Act: -0.0000

Data Index: 548623 (The Pile)

Max Activating Token Index: 129

Text #13

Max Range: 0.7764. Min Range: -0.7764

Max Act: 0.4936. Min Act: -0.0000

Data Index: 1664521 (The Pile)

Max Activating Token Index: 132

Text #14

Max Range: 0.7764. Min Range: -0.7764

Max Act: 0.4831. Min Act: -0.0000

Data Index: 73061 (The Pile)

Max Activating Token Index: 917

Text #15

Max Range: 0.7764. Min Range: -0.7764

Max Act: 0.5017. Min Act: -0.0000

Data Index: 1466705 (The Pile)

Max Activating Token Index: 205

Text #16

Max Range: 0.7764. Min Range: -0.7764

Max Act: 0.4947. Min Act: -0.0000

Data Index: 474277 (The Pile)

Max Activating Token Index: 243

Text #17

Max Range: 0.7764. Min Range: -0.7764

Max Act: 0.4777. Min Act: -0.0000

Data Index: 356985 (The Pile)

Max Activating Token Index: 73

Text #18

Max Range: 0.7764. Min Range: -0.7764

Max Act: 0.4721. Min Act: -0.0000

Data Index: 306184 (The Pile)

Max Activating Token Index: 751

Text #19

Max Range: 0.7764. Min Range: -0.7764

Max Act: 0.4898. Min Act: -0.0000

Data Index: 711282 (The Pile)

Max Activating Token Index: 881