Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 1962 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 2.6339. Min Range: -2.6339

Max Act: 2.6339. Min Act: -0.0000

Data Index: 211933 (The Pile)

Max Activating Token Index: 214

Text #1

Max Range: 2.6339. Min Range: -2.6339

Max Act: 2.5803. Min Act: -0.0000

Data Index: 1746135 (The Pile)

Max Activating Token Index: 830

Text #2

Max Range: 2.6339. Min Range: -2.6339

Max Act: 2.4503. Min Act: -0.0000

Data Index: 1595360 (The Pile)

Max Activating Token Index: 405

Text #3

Max Range: 2.6339. Min Range: -2.6339

Max Act: 2.4510. Min Act: -0.0000

Data Index: 389875 (The Pile)

Max Activating Token Index: 509

Text #4

Max Range: 2.6339. Min Range: -2.6339

Max Act: 2.4115. Min Act: -0.0000

Data Index: 1022591 (The Pile)

Max Activating Token Index: 926

Text #5

Max Range: 2.6339. Min Range: -2.6339

Max Act: 2.5375. Min Act: -0.0000

Data Index: 805335 (The Pile)

Max Activating Token Index: 415

Text #6

Max Range: 2.6339. Min Range: -2.6339

Max Act: 2.3758. Min Act: -0.0000

Data Index: 884483 (The Pile)

Max Activating Token Index: 592

Text #7

Max Range: 2.6339. Min Range: -2.6339

Max Act: 2.2376. Min Act: -0.0000

Data Index: 182524 (The Pile)

Max Activating Token Index: 905

Text #8

Max Range: 2.6339. Min Range: -2.6339

Max Act: 2.3439. Min Act: -0.0000

Data Index: 1558141 (The Pile)

Max Activating Token Index: 918

Text #9

Max Range: 2.6339. Min Range: -2.6339

Max Act: 2.2418. Min Act: -0.0000

Data Index: 878053 (The Pile)

Max Activating Token Index: 504

Text #10

Max Range: 2.6339. Min Range: -2.6339

Max Act: 2.4520. Min Act: -0.0000

Data Index: 1083325 (The Pile)

Max Activating Token Index: 602

Text #11

Max Range: 2.6339. Min Range: -2.6339

Max Act: 2.2558. Min Act: -0.0000

Data Index: 1599079 (The Pile)

Max Activating Token Index: 715

Text #12

Max Range: 2.6339. Min Range: -2.6339

Max Act: 2.2174. Min Act: -0.0000

Data Index: 1645867 (The Pile)

Max Activating Token Index: 703

Text #13

Max Range: 2.6339. Min Range: -2.6339

Max Act: 2.3032. Min Act: -0.0000

Data Index: 493267 (The Pile)

Max Activating Token Index: 539

Text #14

Max Range: 2.6339. Min Range: -2.6339

Max Act: 2.0785. Min Act: -0.0000

Data Index: 520253 (The Pile)

Max Activating Token Index: 658

Text #15

Max Range: 2.6339. Min Range: -2.6339

Max Act: 2.0477. Min Act: -0.0000

Data Index: 1467084 (The Pile)

Max Activating Token Index: 21

Text #16

Max Range: 2.6339. Min Range: -2.6339

Max Act: 2.1547. Min Act: -0.0000

Data Index: 1709777 (The Pile)

Max Activating Token Index: 718

Text #17

Max Range: 2.6339. Min Range: -2.6339

Max Act: 2.0274. Min Act: -0.0000

Data Index: 827148 (The Pile)

Max Activating Token Index: 987

Text #18

Max Range: 2.6339. Min Range: -2.6339

Max Act: 2.1349. Min Act: -0.0000

Data Index: 1301993 (The Pile)

Max Activating Token Index: 907

Text #19

Max Range: 2.6339. Min Range: -2.6339

Max Act: 2.1459. Min Act: -0.0000

Data Index: 1540212 (The Pile)

Max Activating Token Index: 625